Preparing Clean Views of Data for Data Mining

Good data preparation is a key prerequisite to successful data mining [P99]. Conventional wisdom suggests that data preparation takes about 60 to 80% of the time involved in a data mining exercise [R97]. There have been good reviews of the problems associated with data preparation [F97, HS98 and MS97]. However the data cleaning aspect of data preparation is regarded as involving major human input and often has been neglected in practice. This paper reports work undertaken in support of a data mining programme at Rutherford Appleton Laboratory (RAL). It proposes a Clean Views Model for data cleaning. The action taken to clean data depends on the business purpose of the analysis. The cleaning action also depends on what you believe about the data expressed in the business rules that the data need to satisfy. These rules are expressed in a validator function . What you believe about the data can change during the data mining process and also business purposes change through time. This means that data cleaning should be described as an iterative exercise rather than a one-off procedure. We contend that the original, unclean data should be retained and that the cleaning produces a particular view, a Clean View, for a particular validation function. We have developed the Clean Views approach after looking at several datasets. We report some work on the flow of funds data associated with transitional economies [Ba99]. These data were generated by the central banks of Eastern European countries and are used to assess their movement towards a market economy. We also report work done on data in an international mailing list concerned with a survey of alumni of business schools who have graduated as Masters of Business Administration.